Surgical display

ABSTRACT

Disclosed herein are visualization systems, methods, devices and database configurations related to the real-time depiction, in 2 D and 3 D on monitor panels as well as via 3 D holographic visualization, of the internal workings of patient surgery, such as patient intervention site posture as well as the positioning, in some cases real time positioning, of an object foreign to the patient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This document claims priority to U.S. Prov. Ser. No. 63/150,761, filedFeb. 18, 2021, the disclosure of which is hereby incorporated byreference in its entirety; to U.S. application Ser. No. 17/331,957,filed May 27, 2021 which issued as U.S. Pat. No. 11,273,003 on Mar. 15,2022, the disclosure of which is hereby incorporated by reference in itsentirety, to U.S. application Ser. No. 17/591,053, filed Feb. 2, 2022,which issued as U.S. Pat. No. 11,478,327 on Oct. 25, 2022, thedisclosure of which is hereby incorporated by reference in its entirety,and to U.S. application Ser. No. 17/948,861 filed Sep. 20, 2022, thedisclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

Invasive surgery is a balance between a need to access an internalregion and a desire to minimize disruption at that region to no morethan necessary to effect the surgical goal. This balance is often skewedby the need of a surgeon to see the region that is the target of thesurgery, and often to deliver a surgical insert to the region. Thus,surgery is in some cases a trade-off between gaining access, seeingone's target and actually effecting change at the site of theintervention.

A technology that reduces need to visualize the insertion site may shiftthis balance, so that surgeries may be completed with reduced impact onthe patient, and without negatively impacting a surgeon's ability toview the site of intervention.

SUMMARY

Imaging and visualization systems are disclosed. The systems enableusers to generate, view and manipulate three dimensional depictions ofcomplex systems such as patient surgical intervention sites, and to viewobjects such as surgical inserts in situ at the patient interventionsites. The systems comprise one or more of a 2 dimensional display, aholographic 3 dimensional light field display, a data storage processorconfigured to retain configurational data for an object, such as anobject representative of a surgical insert, and to receive data fromwhich a patient surgical intervention site image may be generated, and aprocessor to generate such an image. Positioning the object such as asurgical insert is facilitated by position sensors such as passiveposition sensors that may be affixed to an object such as a surgicalinsert. The system further comprises an interface that obtains positioninformation from the position sensors, and a processor capable ofadjusting an image in response to position information received from theinterface, for example by repositioning the object such as a surgicalinsert relative to the patient intervention site image. The image isadjusted, in some cases, in real time concurrently with surgery so as topresent an image depicting a surgical insert in the posture which itactually is assuming ion the patient. In other cases the images areadjusted by a user pursuant to surgical planning prior to initiatingsurgery. Images are in some cases manipulated by a user through use of ahands-free sensor, so that a surgeon may, in real time, view andmanipulate an image of a patient surgical intervention site that depictsthe real time posture of a surgical insert within the patient, allwithout needing to touch a monitor, keyboard or touchpad, therebykeeping the surgeon's hands free for patient interaction. Images arereadily observed by the ‘naked eye’ unaided by virtual reality optics orother headgear, further facilitating surgeon performance. Furthermore,images are presented such that multiple individuals may concurrentlyview an image directly. Methods related to use of these systems, and touse of images such as those that could be generated by these systems,are similarly disclosed.

Disclosed herein are imaging systems suitable for visualization ofsurgical procedures. Some such systems comprise one or more of: a 2 Ddisplay; a holographic light field display; a data storage processorconfigured to retain configurational data for an object; a plurality ofpassive position sensors; an interface that obtains position informationfrom the passive position sensors; and a processor capable of adjustingan image in response to position information received from theinterface. The image is ‘malleable’ in that it may be adjusted withoutreplacing the configurational data, rather than being replaced by asecond image drawn from a second set of configurational data capturedafter a surgical manipulation. Consequently image adjustment is madeefficiently, rapidly, and without the need for configurational datacapture equipment to be present during surgery.

In some aspects, these image systems comprise one or more of thefollowing: a data storage processor configured to receive patient bodystructural data, such as wherein the patient body structural data iscompatible with 2D display, wherein the patient body structural data iscompatible with display on a holographic light field display, whereinthe data storage processor converts 2D patient body structural data intodata suitable for display on a holographic light field display, whereinthe data storage processor converts data suitable for display on aholographic light field display into patient body structural datacompatible with a 2D display, wherein the imaging system displays thepatient body structural data, wherein the imaging system displays thepatient body structural data on the 2 D display, wherein the imagingsystem displays the patient body structural data on the holographiclight field display, wherein the imaging system concurrently displaysthe patient body structural data on the 2 D display and on theholographic light field display, wherein the imaging system displays thepatient body structural data and the configurational data for an object,wherein the object is a surgical implant object, wherein the surgicalimplant object is a spinal implant object, wherein the surgical implantobject is a cranial implant object, wherein the object is a surgicaltool, wherein the surgical tool is a spinal surgery tool, wherein thesurgical tool is a cranial surgery tool, wherein the surgical tool is acardiac surgery tool.

In some aspects, location of the object relative to an image of thepatient body structural data is informed by at least one passiveposition sensor of the plurality of passive position sensors. Theposition information in some cases exhibits one or more of thefollowing: it is obtained from a passive position sensor on the object,is obtained from a passive position sensor on a patient, is obtainedfrom a passive position sensor on a user, or is obtained from a passiveposition sensor on a hand of a user. A partial list of surgical objectsincludes, for example, a spinal insert, a spinal surgical tool, acranial insert, a cranial surgical tool, a surgical insert, a surgicaltool, a stent, a brace, a screw, a mechanical device, a pacemaker, anartificial bone, an artificial organ, an artificial tissue, and atransplant. In cases where the surgical object is intended for long terminclusion in a patient, the position sensor is optionally removed priorto resolution of surgery.

In some aspects, adjusting an image in response to position informationreceived from the interface comprises rotating the image. The imageadjusting in some cases exhibits one or more of the following: adjustingan image in response to position information received from the interfacecomprises spatially translating the image, adjusting an image inresponse to position information received from the interface comprisesadjusting scale of the image, adjusting an image in response to positioninformation received from the interface comprises adjusting the positionof the object relative to the patient body structural data, adjusting animage does not require receipt of data from a keyboard, adjusting animage does not require receipt of data from a touchpad.

In some aspects, the system does not require imaging glasses.

In some aspects, the system does not require imaging headgear.

Also disclosed herein are methods of visualizing a patient surgery. Somesuch methods comprise one or more of the following elements: displayinga holographic depiction comprising a patient surgical region and asurgical object, and moving the surgical object relative to the patientsurgical region in the holographic depiction.

In some aspects, the methods comprise one or more of the followingelements: displaying does not require user headgear; displaying a 2 Ddepiction comprising the patient surgical region and the surgicalobject; moving the surgical object relative to the patient surgicalregion in the holographic depiction does not require contacting akeyboard; moving the surgical object relative to the patient surgicalregion in the holographic depiction does not require contacting atouchpad; moving the surgical object relative to the patient surgicalregion in the holographic depiction comprises receiving passive positiondata; moving the surgical object relative to the patient surgical regionin the holographic depiction comprises physically moving a surgicalobject; the passive position data comprises patient surgical regionpassive position data; patient surgical region passive position datacomprises data for a region that is not visibly accessible duringsurgery; the passive position data comprises surgical object passiveposition data; the passive position data comprises user passive positiondata; the surgical object comprises a spinal insert; the surgical objectcomprises a spinal surgical tool; the surgical object comprises acranial insert; the surgical object comprises a cranial surgical tool;the surgical object comprises a surgical insert; the surgical objectcomprises a surgical tool; the method is performed prior to patientsurgery; the method is performed concurrently with patient surgery; theholographic depiction comprising a patient surgical region is derivedfrom patient MRI data; the holographic depiction comprising a patientsurgical region is derived from patient X-ray data; the holographicdepiction comprising a patient surgical region is derived from patientCT data.

Also disclosed herein are surgical visualization systems, such as thoseused to prepare for or to observe ongoing surgery. Some such systemscomprise one or more of the following elements: a processor configuredto receive patient image data, a file set comprising surgical objectdata, a 2 D patient data depiction window, a 3 D patient data depictionwindow, and a selection option that allows a user to introduce thesurgical object data into the patient image data, thereby generatingdata so as to depict a 2 D image of the surgical object data introducedinto the patient image data and a 3 D image of the surgical object dataintroduced into the patient image data.

In some aspects, the systems comprise one or more of the followingelements: the patient image data comprises 2 D slices; the patient imagedata comprises CT data; the patient image data comprises MRI data; thepatient image data comprises X-ray data; the processor converts the 2 Dslices into a volumetric 3 D rendering; the surgical object datacomprises at least one of a medical pedicle screw, a cervical interbody,a lumbar interbody, a cervical plate, a lumber plate and a correspondingscrew; the surgical object data comprises at least one surgical device;further comprising a position sensor system; the position sensor systemcomprises at least one passive sensor and at least one passive sensortracking device; at least one passive sensor is attached to a surgicalinstrument; a reference bracket system comprising a component attachedto a patient that correlates the patient to the patient image data; thesurgical instrument is displayed relative to the patient image data;concurrent 2 D depiction and 3 D depiction convey surgical instrumentdepth relative to the patient image data; comprising a holographicdisplay that generates a 3 D holographic image; the holographic displaydepicts at least one of the patient image data, the surgical object dataand the surgical instrument in the 3D holographic image; at least one ofthe patient image data, the surgical object data and the surgicalinstrument is depicted in a contrasting color; a user eye monitor thattracks user eye motion and repositions the 3D holographic image inresponse to user eye movement; the system does not require user headgearfor a user to view the 3 D holographic image; a touch free motioncontroller, such as one that detects user hand position; a change inuser hand position results in a change in position of the 3 Dholographic image.

In some aspects the system allows hands-free manipulation of the 3 Dholographic image

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a surgeon using a system as described herein.

FIG. 2 depicts a surgeon using a system as described herein.

DETAILED DESCRIPTION Introduction

Disclosed herein are visualization systems, methods, devices anddatabase configurations related to the real-time depiction, in 2 D and 3D on monitor panels as well as via 3 D holographic visualization, of theinternal workings of patient surgery, such as patient intervention siteposture as well as the positioning, in some cases real time positioning,of an object foreign to the main image (the patient in most cases), forexample a surgical insert, tool or both surgical insert and tool,relative to a patient intervention site, or of a subset of the mainimage relative to the remainder of the main image. Depictions are madeso as to reduce a surgeon's need to directly see ‘with own eyes’ anintervention site, such that in some cases surgery is performed withouta surgeon needing to see an intervention site directly, as well as tofacilitate preparation of surgery.

The depictions generated hereby are readily manipulated, such that asurgeon may adjust the scale, orientation or other characteristics of a2 D or 3 D monitor depiction or a 3 D holographic depiction of anintervention site or a surgical insert or surgical tool deposed therein.Manipulation is touch free in some cases, such that a user maymanipulate a depiction without physically contacting a device, such as akeyboard, monitor or touchpad. Touch-free manipulation facilitatessurgical economy and speed, by reducing the risk of surgeon handcontamination, and therefore the time spent ensuring hand sterilization.Touch free manipulation is accomplished through detection of user handorientation in space, for example, or by monitoring user eye movementsuch that eye positioning or movement is used to direct depictionmanipulation. Touch free manipulation may also be accomplished throughverbal instruction by a user, such as to a voice recognitionfunctionality. Alternate routes of manipulation, such as through footpedals or other user motion, are also consistent with the disclosureherein. Also, in some alternate cases manipulation is effected using aconventional input such as a keyboard, mousepad or touchpad.

The technology herein is described in the context of surgicalinterventions, which are a preferred embodiment. Nonetheless, one willobserve that the disclosure may be applied to a number of scenariosoutside of surgery where a primary image is depicted as a 3 Dholographic image, and a secondary image or a portion of the primaryimage is manipulated relative to the primary image or remainder of theprimary image, such that an observer or user is able to see the spatialconsequences of such a manipulation.

Data

The visualization systems, methods, devices and database configurationsdisclosed herein are often compatible with a broad range of datasources. Data are often provided as two dimensional slices of a patientor patient intervention site, which are computationally converted into 3D configurations compatible with depiction of 2 D slice data as 3 Dimages on a monitor, or in a 3 D holographic projection, or both.Datasets, either received or converted 3 D image datasets, are often‘malleable.’ in that the 2 D and 3 D images derived therefrom areadjustable, particularly in light of positional data such as thatobtained through passive positional markers applied to a patient, suchas a patient from which the 2 D slice data is obtained. That is, 2 D and3 D images that are derived from a dataset, presented on screen orholographically, may be adjusted in light of positional data such asthat obtained through passive positional markers applied to a patientrather than merely generating a new dataset subsequent to adjusting apatient intervention site and generating a new 2 D or 3 D image from thenew data set. As dataset generation is time consuming, computationallyintensive and in some cases demanding as to image collection deviceneeded to collect the data, adjusting an image generated from theoriginal dataset in light of passive positional markets represents asubstantial improvement in speed and efficiency of image creation.Additionally, by not requiring re-imaging of a patient using an imagingdevice during surgery, the surgery environment is made substantiallysimpler, as the imaging device need not be present during surgery. Thiscreates more room during surgery and obviates the need to ensure that amimaging device be sterilized prior to surgery. Separately, data areprovided corresponding to one or more surgical insert, such as surgicalinserts to be employed in a surgical procedure. Similarly, data are alsoprovided corresponding to one or more surgical instruments, such as asurgical instrument used to deliver a surgical insert to a surgicalintervention site of a patient.

Positioning of a surgical insert or a surgical instrument, or both,relative to a patient surgical intervention site is reflected in datasite depictions, such as patient images, into which images of thesurgical insert or a surgical instrument, or both are introduced andpresented in position and orientation reflecting that of the actualsurgical insert or a surgical instrument, or both, relative to theactual patient intervention site. Often, this relative positioning isinformed by positional markers, such as passive positional markers.Detection of the positional markers such as the passive positionalmarkers, is informative of the relative position of the surgicalinstrument or surgical insert or both relative to the patient insertionsite, and of changes in the patient positioning relative to the patientpositioning in the original dataset. In preferred embodiments, depictionof this positioning of one or more of patient, surgical device andsurgical implant, and changes in this positioning, is effected in realtime, such that the image accurately depicts the actual state of theunderlying objects and patient in real time, so as to be beneficial to asurgeon performing such a procedure.

A number of 2 D slice data acquisition methods, and datasets, producedthereby, are consistent with the present disclosure. Datasets arepreferably convertible into 3 D configurations for monitor andholographic display. Exemplary datasets from which 3 D images arederived include Magnetic Resonance Image (MRI) data sets and computerizetomography (CT) scan data sets, although other data sources capable ofproviding sectioned data slices such as ultrasound, X-ray, fluoroscopy,angiography, mammography, breast tomosynthesis, PET (positron emissiontomography), SPECT (single photon emission computed tomography),endoscopy, microscopy, whole slide imaging, OCT (optical coherencetomography) are also suitable in some cases.

Data is often configured into Digital Imaging and Communications inMedicine (DICOM) format, such as DICOM 3.0, prior to delivery to adevice as disclosed herein. However, the present disclosure does not inall cases require DICOM 3.0 or even alternate DICOM formatted data.Alternate formats, such as DICONDE (Digital Imaging and Communication inNondestructive Evaluation) and other formats are compatible with someembodiments of the systems, devices and methods of the presentdisclosure.

Generally, data which is obtained from 2 D slices of a subject is usedmost often, but any data that can be configured into images of a patientis compatible with some embodiments of the present disclosure.

Similarly, data which is already configured into a format conducive to 3D or 3 D holographic display is also compatible with the presentdisclosure. Surgical implants and surgical tools, such as tools forsurgical implant delivery, are often configured into a format conduciveto 3 D or 3 D holographic display rather than being presented as 2 Dsections. Data already configured into 3 D format is also compatiblewith some embodiments of the present disclosure.

Computation

The visualization systems, methods, devices and database configurationsdisclosed herein are often reliant upon a computational system able toconvert 2 D data into 3 D image datasets, and to adjust those datasetsin response to additional input such as positional data regardingsurgical insert or surgical instrument position data, or changes inpatient posture or positioning. This additional input often arises fromthe detection of positional markers, such as passive positional markers,affixed to one or more of a surgical insert, a surgical instrument, or apatient.

Computational systems as disclosed herein often allow for the loading ofimageable data (for example, DICOM medical image data such as CT and MRIscans, or other types as discussed herein). Data may be obtained from aremote source, wirelessly, from a drive or directly from PACS in ahospital's network. Data that is not already formatted for 3 D depictionmay be subjected to one or more of segmentation, registration, andvisualization.

Computational devices are often configured to translate the 2D slices ofDICOM or other 2 D data formats into a volumetric 3D rendering so thatthe user can see both the 2D view of the data subject such as thepatient's anatomy as well as a 3D view.

Computational devices are often also pre-loaded or subsequently loadedwith digital files of a surgical implant or implants and optionallysurgical tools such as surgical implant delivery tools, for exampleMedical pedicle screws, cervical and lumbar interbodies, as well ascervical and lumber plates and corresponding screws. Other surgicaltools are consistent with the disclosure herein and are alsocontemplated. An incomplete list of surgical tools or other objects tobe imaged in concert with patient data includes, for example, a stent, abrace, a screw, a mechanical device, a pacemaker, an artificial bone, anartificial organ, an artificial tissue, or a transplant tissue or organ,for example.

The user can pick specific implants and specific sizes, for example froma dropdown menu or other selection interface, and load them into theviewer with the patient anatomy. Alternately, specific implants, toolsor other surgical devices may be measured so as to generate datasetssuitable for their depiction in one or more of a holographic, 2 D or 3 Dimage as contemplated herein. Adding this material into a depictionfacilitates the evaluation of various implants, tools or other surgicaldevices relative to each patient's anatomy, so as to help in pre-surgeryimplant selection or surgery strategy, for example. In some cases, inneighboring windows, the user sees both the 3D view of the implants andanatomy as well as a 2D view of the implants and anatomy simultaneously,as an alternative to or in combination with a 3 D holographic image.

Computational devices are often integrated with a position sensorapparatus such as the Polaris Vega VT Position Sensor by NorthernDigital for surgical navigation, a Holographic Light Field Display forholographic visualization, and a Leap Motion controller for touch-freemanipulation of the anatomy.

A beneficial feature of some computational systems disclosed herein isthat adjustments to datasets, or to images depicting that data, are madein real time, such that a surgeon using the visualization systems,methods, devices and database configurations disclosed herein is able tosee an up to date, volumetric image of a surgical intervention site, inmultiple views on screen and in holographic projection, and to adjustthe perspective or angle of view of this volumetric image, as thesurgical intervention is taking place. This represents a substantialimprovement over some existing technologies, where a dataset is replacedand re-collected subsequent to an adjustment to a surgical interventionsite. In the present disclosure, positional information is gathered andused to adjust the image in light of a previously collected dataset.This eliminates the need for elaborate data collection instruments to bepresent in the surgery environment, and eliminates the need to replaceentire datasets in an ongoing process throughout surgery. In its stead,data regarding the location of positional markers are gathered, andthese data are used to modify the underlying patient dataset or theholographic image produced therefrom. This approach is substantiallyless computationally intensive, can be effected quickly with lesscomputational burden, and requires a substantially smaller footprint inthe surgery environment, which in turn facilitates a surgicalenvironment that is more readily sterilized because large imagingmachinery need not be present and re-used in multiple surgeries. Moregenerally, the surgical environment is less cluttered, promoting a lessdistracting environment for a medical professional or professionals tooperate.

To facilitate this real-time depiction, the image can often bemanipulated in a hands-free manner, for example by a device orfunctionality that detects hand movements or eye movements so as todirect the depiction of the image. The real-time depiction of someembodiments, alone or in combination with the hands-free control of manyembodiments, facilitates faster, more efficient surgery because thesurgeon is able to see the surgical intervention site, surgical insertand perhaps surgical insertion tool from multiple angles without needingto actually directly see into the patient, and without needing to riskcontamination to hands by physically touching the device to adjust theimage.

Alternately or in combination, the visualization systems, methods,devices and database configurations disclosed herein may be used priorto surgery, for example as part of a surgical planning process. Such aprocess may comprise computational prediction of the impact or effect ofa surgical insert or surgical tool on a surgical intervention site, soas to facilitate surgical insert selection and planning of the specificapproach and procedure for delivery of an insert to a surgicalintervention site. In some cases the image is computationally adjustedso as to simulate the effect of the surgical tools on the patient.

Alternately or in combination, the visualization systems, methods,devices and database configurations disclosed herein may be usedpursuant to instruction, such as surgical instruction pursuant tosurgical or medical education. Such a process may comprise real timebroadcasting or broadcasting of a recorded series of surgical events orholographic images, such as to an audience. Such an audience maycomprise students and in some cases an instructor, who may be present orremote to the location of the surgery. In preferred embodiments studentsare able to observe one or more of a planned surgical procedure or areal time depiction of a surgical procedure through observing aholographic image, alone or in combination with a 2 D or 3 D or both 2 Dand 3 D depiction of a surgical intervention site.

Students are in some cases able to manipulate the holographic or otherimages, for example using a hands free approach as described herein orthough interaction via a touchpad, keyboard, mouse, or otherconventional computer interface. In particular, as preservation of asterile environment is less critical to some instruction approaches,there may be broader flexibility as to how one interacts with an imagepursuant to instruction. Alternately, in cases where simulation of asurgical environment is a priority, one may take steps or precautions ininstruction that are comparable to those steps or precautions takenpursuant to actual surgery.

Adjustments to an image, either as done by a user so as to gain adesired perspective or line of site for a surgical intervention site orother site of a subject or resulting from data gathered by positionalmarkers so as to adjust the image to reflect a surgical interventionsite posture subsequent to a surgical action, for example, are in somecases recorded. Such a recording may be stored and replayed pursuant tomedical instruction or to other observation of how a surgical procedureprogressed.

Monitors

The visualization systems, methods, devices and database configurationsdisclosed herein often comprise one or more than one monitor. A monitoris used to depict data images, in either 2 D, 3 D or both 2 D and 3 D,alone or in combination with other data relevant to a surgicalprocedure, such as heart rate, breath rate, muscle activity, brainwaveactivity, or other patient data or data relevant to successful surgery.Data images are in some cases presented in real time, such that asurgeon is able to view, on a monitor, a patient surgical interventionsite as it is being manipulated pursuant to surgery, for example throughinsertion of a surgical implant, manipulation using a surgicalinstrument, or both.

Alternately or in combination, data images may be used prior to surgery,for example as part of a surgical planning process. Such a process is insome cases facilitated by the 2 D and 3 D visualization on a monitor ofany of a surgical intervention site, a surgical insert, and a surgicalinsertion tool, so as to facilitate surgical insert selection andplanning of the specific approach and procedure for delivery of aninsert to a surgical intervention site.

Alternately, some embodiments of the methods and systems herein do notrely upon all of 2 D and 3 D monitors and holograms. Instead, thesesystems rely upon some subset of these display options, such as 2 D and3 D monitors, 2 D monitors and holograms, 3 D monitors and holograms, or2 D monitors alone, 3 D monitors alone, or holograms alone.

Holograms

The visualization systems, methods, devices and database configurationsdisclosed herein often comprise one or more devices for the projectionof holographic images, or the use of images so projected. Holographicimages are used alone or in combination with images portrayed onmonitors, as described above and elsewhere herein.

Holographic images can be projected to scale and directly in front of auser, such that they give the sense of a patient's anatomy actuallyhovering in air, and may be scaled, rotated or translated to changetheir position, or otherwise manipulated so as to facilitate viewing ofa depiction of a patient anatomy. Such projections facilitate concurrentviewing by a plurality of users or medical professionals, such asdepicted in FIG. 2 . Holographic images may have a number of sizeranges, such as at least, at most, about or exactly, 5, 6, 7, 8, 9, 10,11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,47, 48, 49, 50 or more than 50 inches, or any size in a range from orless than or exceeding 0.1×, 0.2×, 0.5×, 1×, 2×, 5× or 10× relative toan surgical target actual size.

Holographic images are often generated from data which as describedabove are often provided as two-dimensional slices of a patient orpatient intervention site, which are computationally converted into 3 Dconfigurations compatible with depiction of 2 D slice data as 3 D imageson a monitor, or in a 3 D holographic projection, or both. Alternately,some holographic images are generated from data that is alreadyconfigured for 3 D depiction or from other data types.

Holographic images consistent with the disclosure herein may comprisedepictions arising from patient data alone, or both patient data anddata relating to a surgical insert, a surgical tool, such as fordelivery of a surgical insert, or both a surgical insert and a surgicaltool, such as for delivery of a surgical instrument.

Holographic images may arise from data that is informed by positionaldata of a patient, or both patient data and data relating to a surgicalinsert, a surgical tool, such as for delivery of a surgical insert, orboth a surgical insert and a surgical tool, such as for delivery of asurgical instrument. Often, positional data is obtained through the useof positional markers, such as passive positional markers, on one ormore objects corresponding to or depicted in data stored on the deviceas disclosed herein.

The data depicted holographically is informed by positional markers suchthat the data images are adjusted, optionally in real time, to reflectthe actual positions of the patient surgical intervention site, surgicalinsert or surgical tool, pursuant to surgery. By depicting the surgicalenvironment in real time, holographically, and from an angle orperspective and scale selected by a user such as a surgeon, one is ableto visualize a surgical environment without actually directly viewingthe surgical intervention site in the patient. Importantly, adjustmentsto a holographic depiction do not require that a data set be replaced orregathered. Rather, the holographic is malleable in that it may beadjusted relative to an initial or previous image generated fromunderlying data, without the underlying dataset being replaced with asecond data set gathered de novo pursuant to surgery. The holographicimage may be adjusted in light of positional information obtained frompositional markers, or alternately the data may be modified in light ofpositional markets and the image regenerated from the data. However, theunderlying data need not be replaced in an ongoing process throughout aprocedure. This malleability, either of the image relative to an initialdataset or of the dataset itself in light of positional information,allows or facilitates real time adjustment of an image, so as tofacilitate its use during surgery.

Additionally, by not requiring ongoing re-gathering of patient positiondata pursuant to surgery, one eliminates the need for a patient datacollection device (other than positional markers) to be present in asurgical environment. This results in a less crowded and more readilysterile surgical environment.

Alternately, the systems and methods herein do not in all cases precludeiterative data gathering throughout a process, as iteratively gathereddata may be projected and manipulated, such as through a hands-free orconventional input approach, consistent with the remainder of thedisclosure herein.

As discussed above, the perspective, scale or orientation of theholographic image may be changed, such as through a hands-freeinterface. Such an interface may use, for example, hand positioning oreye movement to specify changes in image depictions. Hands-feeadjustments to an image facilitates real time employment of thevisualization systems, methods, devices and database configurationsdisclosed herein in real time in surgery.

Alternately or in combination, the visualization systems, methods,devices and database configurations disclosed herein may be used priorto surgery, for example as part of a surgical planning process, or priorto concurrent with or subsequent to surgery as part of an instructionalprocess. Such a process is in some cases facilitated by the holographicvisualization of any of a surgical intervention site, a surgical insert,and a surgical insertion tool, so as to facilitate surgical insertselection and planning of the specific approach and procedure fordelivery of an insert to a surgical intervention site.

Holographic images of the present disclosure are optionally mono-coloror multicolored. Multicolored holograms may display patient image andsurgical insert image or surgical tool in contrasting colors.Holographic images may differentially color various parts of a patientimage, such as tissues that are stretched, contorted or otherwisesubjected to pressure, stress, torsion or other perturbation pursuant toor prior to surgery. Differential coloring is in some cases adjustable,such that a user may select a portion or portions of a holographic imageto highlight in one or more distinct colors.

Notably, many embodiments of the holographic projection devicesdisclosed herein project holographic images that are readily observed bythe naked eye of some users. That is, many embodiments of theholographic projections consistent with the disclosure herein do notrequire personal optical equipment or headgear, such that holographicimages projected thereby are readily observed without the mental orocular fatigue that is in some cases associated with holographicheadgear. Particularly when used in combination with surgicalinterventions that last for a substantial period of time, such as hours,the reduced mental and ocular fatigue is a significant benefit to someusers. Furthermore, these projections may be viewed in some cases bymultiple users concurrently, as depicted in FIG. 2 .

Passive Detection

The visualization systems, methods, devices and database configurationsdisclosed herein often present image data not only as originallyreceived but also as modified to reflect changes in patient surgicalintervention site position or posture (for example to accommodate asurgical implant or a surgical tool such as a surgical implant deliverydevice), as well as to reflect the introduction or change in position ofa surgical implant or a surgical tool such as a surgical implantdelivery device at a patient surgical intervention site.

Data indicative of such a change in position or posture of a patient orobject at a site of surgical intervention is often collected through theuse of a detection device or system Some such detection systems comprisedelivering or affixing a detectable device to an object or solidmaterial to be monitored such as a surgical implant or a surgical toolsuch as a surgical implant delivery device, or even to a patient. Somedetectable devices actively send position information to an interfacewith a computation module of a system as disclosed herein, so as tomodify an image or images in light of changes in positional information.Alternately, in some preferred embodiments, passive detectable devicesare used to mark an object or solid material to be monitored such as asurgical implant or a surgical tool such as a surgical implant deliverydevice, or even to a patient. In these embodiments, passive detectiondevices such as tags work in combination with a monitoring device, suchas one that actively detects passive detection tags so as to obtainadditional or modifying positional information sufficient, incombination with previously obtained positional data or on its own, tomodify or create new images relevant to a patient surgical interventionsite. Detectable devices are variously attached to a target via anadhesive, structurally built into an or otherwise attached to an objectsuch as a surgical insert or surgical tool such as surgical insertdelivery device or adhered to a patient.

Passive detection devices are in some cases used to facilitate real-timevisualization of a patient surgical intervention site, the posture of asurgical insert or surgical insert delivery tool or all of the above. Bydetecting changes in patient positioning or the posture of a surgicalinsert or surgical insert delivery tool or both, the systems and methodsherein in some cases allow real time visualization on a 2 D monitor of a2 D depiction or a 3 D depiction, or via a holographic projection, thevisualization of a 3 D holograph of a surgical intervention site andsurgical instrumentation in situ.

Passive detection devices as disclosed and contemplated herein allow oneto make an image or patient data set to be ‘malleable’ such that it maybe adjusted rather than throwing out or replacing a data set in itsentirety subsequent to a surgical site manipulation that may change theposition of a surgical instrument or patient sot or hard tissue.

Image Manipulation

The visualization systems, methods, devices and database configurationsdisclosed herein often present image data in a form that is readilymanipulated by a user rather than being or appearing inflexible orrigid. For example, 2 D images or 3 D images on monitors or 3 Dholographic images are readily manipulated in some cases, such that auser may rotate, zoom in or out, or otherwise manipulate the posture orpresentation of an image. Such manipulation facilitates a user'sexamination of a patient surgical intervention site, alone or incombination with or adjusted to accommodate a surgical insert or asurgical tool for insertion of a surgical insert, or both.

Manipulation is effected through any of a number of approaches. Someembodiments rely upon detection of user movement, such as hand gestures.Gestures are performed at a detection site, such as over a controller,and the image is adjusted in response to the hand movement. Similarly,some devices rely upon other user movement, such as user eye movement orhead movement, to direct image repositioning or reorienting Contact freeinput is preferred, for example, when an image is observed duringsurgery, such as for real-time depiction of a surgical image, so thatthe user's hands are free for surgical manipulations, and so that theuser's hands are not at risk for desterilization through contact to aninput device.

Alternately, a contact-based manipulation system such as foot pedals canbe used to manipulate an object, as can more conventional approachessuch as touch screens, keyboards, mouse pads or other input. Contactbased input is more favorable when images are observed prior to asurgical procedure, such as in planning a surgical procedure orselecting a surgical insert or surgical tool.

Surgeries

The systems, devices and methods herein are compatible with a broadrange of surgeries and medical interventions Spinal surgeries are aclear lead example, particularly due to the frequent use of surgicalinserts and the inaccessibility of the surgical target region. However,a broad range of alternatives also benefit from the systems, devices andmethods. Some of these procedures similarly comprise delivery of asurgical insert, in some cases to a visually removed target site.However, alternatives, some not even requiring a surgical insert, arealso compatible with the present disclosure. The systems, devices andmethods are compatible with, as a nonlimiting list of examples,interventions in neurology, cranial surgery, brain surgery, such astumor removal, radiology, cardiology, oncology, nuclear medicine,radiotherapy, neurology, orthopedics, obstetrics, gynecology,ophthalmology, dentistry, maxillofacial surgery, dermatology, pathology,transplant surgery, replacement surgery, veterinary medicine, andmedical/clinical photography, among others.

Methods of Patient Visualization

Devices and systems as disclosed herein are compatible with a number ofmethods of patient visualization. Visualization methods allow a surgeonor other medical practitioner to plan a surgical or other intervention,for example by visualizing a region or location targeted for a surgicalor other intervention. Alternately or in combination, one may visualizea patient region during a surgical or other intervention, in some casesin real time Visualization approaches allow one, for example to observethe relative positioning of a surgical insert, surgical tool or evensurgical target such as a tumor, herniated disc or other target,relative to other patient tissue.

Some such methods herein variously comprise depicting a patient region,including a surgical intervention target region in combination with asurgical insert or surgical tool such as a surgical insert deliverytool, and manipulating the image so as to visualize moving the surgicalinsert or surgical tool such as a surgical insert delivery tool relativeto the patient image.

Such manipulation can be part of surgical preparation, whereby anoptimal or desired surgical insert or surgical tool such as a surgicalinsert delivery tool position is selected ahead of a surgery.

Alternately, a surgical insert or surgical tool such as a surgicalinsert delivery tool can be manipulated in an image to reflect theposition of the corresponding object in a patient, in some cases in realtime during surgery. Real time or near real time depiction of a surgicalinsert or surgical tool such as a surgical insert delivery tool duringsurgery allows a surgeon to visualize the surgical insert or surgicaltool such as a surgical insert delivery tool relative to patient tissuewithout directly viewing the patient tissue. A benefit to such a methodis that one does not need to see a patient region, such as a recessed ornot readily seen patient region, without the need to open the region tovisual inspection.

Images are depicted holographically and may additionally oralternatively be presented as 2 D or 3 D images on a conventionalmonitor. A user, or a group of users as seen in FIG. 2 , often is ableto view the image without further technical supporting device such asimage or method-specific headgear. Rather, the image is viewed as onewould view any other image, using ‘naked eye’ or glasses or contacts asone would use routinely. No additional headgear, such as virtual realityor image display headgear, is needed for some embodiments of the methodsdisclosed herein. Alternately, practice of the methods herein does notpreclude the use of headgear such as virtual reality or image displayheadgear, but it is not required for every embodiment.

Often, a patient surgical region and a surgical insert or surgical toolsuch as a surgical insert delivery tool are concurrently displayed as ahologram and as a 2 D depiction or a 3 D depiction on a moreconventional monitor.

As mentioned elsewhere herein, manipulation of an image, for examplerotation, translation or zooming in or out, does not in each caserequire contacting a device. Touch-free manipulation, whereby changes toan image are effected by hand gestures above a detection device, eyemovement or other contact-free movement, is particularly useful when themethod is practiced during a surgery, when a surgeon's hands are sterileand occupied in interacting with a patient. Under these conditions thereis no need to contact a monitor, touchpad, keyboard or other inputdevice.

Alternately, in some cases manipulation is effected by contacting to aphysical device such as a monitor, touchpad, keyboard or other inputdevice. Such manipulation is particularly preferred for situations wherethe method is practiced pursuant to surgical planning rather thanexecution of surgery, although use in surgery is not precluded.

Manipulation variously comprises moving the image in toto or moving aportion of the image relative to a second portion of an image, such aswhen one is fitting a surgical insert relative to an insertion site in asurgical planning.

Manipulation may also comprise moving a portion of an image byphysically moving a component of the image and updating the image, forexample in real time or near real time, to reflect the new position ofthe image component. Such manipulation of the image may comprisereceiving passive or active potion data from a surgical insert orsurgical tool, and may comprise receiving patient passive positionaldata, such that the image is manipulated to reflect a change in patienttissue positioning, for example a change in patient tissue positioningthat may occur in a region that is not easily directly viewed or is notvisible to a surgeon.

The image manipulation may comprise movement of a portion of the imagerepresenting, for example, a spinal insert, a spinal surgical tool, acranial insert, a cranial surgical tool, or a surgical insert, a stent,a brace, a screw, a mechanical device, a pacemaker, an artificial bone,an artificial organ, an artificial tissue, a transplant or a surgicaltool or component more generally or specific to another procedure, suchas a procedure as listed elsewhere herein or in regular use.

The image manipulation is performed prior to patient surgery.Alternately or in combination, manipulation is performed concurrentlywith surgery, such as in real time during surgery.

Images manipulated through the methods herein may be generated from MRIdata, CT scan data, X-ray scan data or any data generation methodmentioned elsewhere herein or known in the medical or imaging field.

Images are viewed in some cases in surgery, such that one is able toview both the image and the actual patient of which the image is arepresentation. That is, methods, devices and systems herein in somecases allow viewing of a holographic, or 2 D or 3 D image of a patientregion, concurrently with or in alternative to viewing the physicalpatient surgery region.

Similarly, methods, devices and systems herein in some cases allowviewing of a holographic, or 2 D or 3 D image of a surgical object aswell as or in alternative to the physical surgical object, such assurgical tool or insert, for example a spinal insert, a spinal surgicaltool, a cranial insert, a cranial surgical tool, a surgical insert, asurgical tool, a stent, a brace, a screw, a mechanical device, apacemaker, an artificial bone, an artificial organ, an artificialtissue, or a transplant.

Alternately, the image is in some cases viewed by an audience remotefrom the surgery, such as an instructional class or even a surgeondirecting a surgery remotely rather than ‘in person’ in the proximity ofthe patient. In these cases the image is viewed by an audience that maynot directly look on the physical patient surgical intervention site,although they may concurrently view a 2 D or 3 D depiction of thesurgical intervention site.

Aside from surgery, the methods herein fund use in other fields as well,such as any field where visualization of multiple components in asingle, adjustable holographic image may be of benefit to a user.Examples include academic medical, dental or veterinary research orinstruction, basic biological research such as microscopy research,engineering, architecture or other field where a manipulatable 3 Dhologram, alone or in combination with a 2 D or 3 D monitor image, mayfacilitate comprehension or development. In such cases, the surgicalimplant or surgical tool is readily replaced by a tool or objectcompatible with the topic or research area of interest. In addition, insome cases the image to be viewed is subjected to more substantialmagnification, such as a magnification of about, exactly, comparable to,or within or exceeding a rage of 2×, 5×, 10×, 20×, 50×, 100×, 200×,500×, 1000× or more than 1000×.

Turning to the Figures, one sees the following.

At FIG. 1 , one sees a surgeon using a system and practicing a method asdisclosed herein. A holographic light field display 101 generates aholographic image 102. Monitor 103 presents a 2 D and a 3 D image of thesubject matter of the holographic image 102. Position sensor 104 detectspassive detection tags affixed to the surgical tool 105, which isintroduced to the patient and is accessing the surgical interventionsite. Both the holographic image 102 and the images depicted on themonitor 103 are adjusted in real time so as to depict the posture of thesurgical tool 105 relative to the patient surgical intervention site. Animage of the surgical tool appears, in real time, as part 106 of theholographic image 102, and as images 107 and 108 on the monitor 103.

FIG. 1 illustrates the following. Firstly, the surgical tool ispresented, in real time, in the holographic image and in themonitor-presented 2 D and 3 D images of the surgical intervention site.Secondly, the surgeon does not have a direct line of sight to thesurgical intervention site, but is able to view it, and the posture ofthe surgical tool relative to the site, via the holographic images 102and 106, and via the monitor 103, presenting tool images 107 and 108.Thirdly, the surgeon performing the surgery does not require virtualreality or other vision supplementing headgear to view the holographicimage 102 including the surgical tool image 106, or to view the monitor103 showing surgical tool images 107 and 108, facilitating surgeryperformance by reducing mental or ocular fatigue.

At FIG. 2 , one sees a surgeon using a system and practicing a method asdisclosed herein. A holographic light field display 201 generates aholographic image 202. A gesture-based interaction sensor 203 detectshand gestures made by the surgeon, and signals a computing module toreconfigure the holographic image 202 in response to the surgeon'stouch-free signals. Position sensor 204 detects passive detection tagsaffixed to the surgical tool, not shown. Notably, the surgeon performingthe surgery does not require virtual reality or other visionsupplementing headgear to view the holographic image 102 or the monitor103, facilitating surgery performance by reducing mental or ocularfatigue. Additionally, the surgeon does not need to physically touch anydevice in order to manipulate the holographic image 202, facilitatingongoing surgical work by reducing risk of contaminating the surgeon'shands. The holographic image 202 is presented as if free-floating abovethe patient such that it may be concurrently viewed by multipleindividuals.

As used herein, the term ‘surgeon’ refers to anyone planning, performingor directing another to perform an invasive intervention or anintervention directed to a patient. Examples of such interventionsinclude spine surgeries comprising surgical inserts but may also includemuch broader instances of interventions such as administration ofradiotherapy or other arguably noninvasive procedures that maynonetheless impact internal tissues or benefit from visualization orimaging of such a target region.

A surgeon is an example of a ‘user,’ which as used herein refers toanyone using a system or device herein or practicing a method herein. Inaddition to a surgeon, a user may comprise an observer of a surgery, aninstructor such as a medical instructor, or any individual observing thesurgical depiction as disclosed herein.

A ‘holographic patient surgical region’ refers to a holographicdepiction of a physical patient surgical region, while a ‘physicalpatient surgical region’ refers to the actual physical site of asurgical intervention, such as patient physical tissue.

Similarly, a ‘physical surgical object’ refers to an actual materialsurgical object, such as a surgical tool or surgical implant or even atransplanted tissue or organ, while a ‘holographic surgical object’refers to a holographic image of such a physical surgical object.Physical surgical objects generally are amendable to tracking using oneor more positional markers, which may in some cases be removed from thesurgical object subsequent to positioning in a patient if the surgeryinvolves permanent or long term retention of the physical object in thepatient.

Analogous to ‘holographic patient surgical regions’ and ‘holographicsurgical objects,’ images of patient surgical regions or surgicalobjects may be generated for depiction on 2 D or 3 D on one or moremonitors.

As used herein, the term ‘about’ a number refers to a rangespanning+/−10% of that number. In reference to a range, the term refersto a larger range spanning from 10% below the lower listed limit of therange to 10% above the upper listed limit of the range.

As used herein, the phrase ‘at least one of’ A, B, and C, refers to setscomprising only A, A and B, or A, B, and C, alone or in combination withunlisted constituents.

As used herein, the term ‘comprising’ is pen ended, in that it does notpreclude additional, unlisted elements.

EXAMPLES

Example 1. A surgeon performs spinal surgery on a patient. The patientis subjected to internal imaging, such that a dataset relating to theintervention site of the spinal surgery is available as a set of 2Dimensional slices that collectively form a 3 Dimensional image of theintervention site.

The surgery comprises insertion of a spinal implant using a surgicaltool. Dataset representations of the spinal implant and surgical toolare similarly available.

Data for the patient, implant and tool are uploaded to a visualizationsystem. The system comprises a computational module configured toreceive and manipulate all of the uploaded data, and to depict the dataon a device monitor as a 2 D and 3 D image, as well as being viewable asa 3 D holographic image, optionally at 1:1 scale or a scale preferred bythe surgeon.

The surgical intervention site is internal to the patient and is noteasily visualized directly but is easily observed on the device monitoras a 2 D and 3 D image, as well as being viewable as a 3 D holographicimage, optionally at 1:1 scale or a scale preferred by the surgeon.

The patient is labeled using passive detection tags, such that changesin patient position relative to the pre-generated dataset are measured.Using these measurements, the 2 D and 3 D monitor depictions, and the 3D holographic image, are adjusted to reflect, in real time, the actualposition of the patient and posture of the patient surgical interventionsite in light of the patient position and local manipulation of theintervention site by the surgeon or others.

The surgical implant and the surgical tool are similarly labeled usingpassive detection tags, such that the position and orientation of thesurgical implant and the surgical tool are measured. Using thesemeasurements, the 2 D and 3 D monitor depictions, and the 3 Dholographic image, are updated to reflect the position and orientationof the surgical implant and the surgical tool relative to the patient.

These updates to the datasets and to the 2 D and 3 D monitor depictions,and the 3 D holographic image, are made in real time, so that thesurgeon is able to see the surgical implant and the surgical tool asthey are positioned relative to the patient's intervention site.

Example 2. The surgeon performs a surgery on a patient as in Example 1The intervention site, surgical implant and surgical tool are readilyobserved through the 2 D and 3 D monitor depictions, and the 3 Dholographic image. As a result, the incision and impact surgicalintervention site sizes are reduced through disruption of less spacethan is sufficient for the surgeon to directly look at all or part ofthe intervention site. Nonetheless, the surgeon is able to visualize theposition of the spinal implant and the surgical tool by observing thesurgical implant and the surgical tool, depicted in their actualpositions relative to the patient intervention site, in real time.

Example 3. The surgeon performs a surgery on a patient as in Example 1or Example 2. The depiction is available to the surgeon as a 2 D screendepiction, a 3 D screen depiction, and as a 3 D holographic image. Allare presented to the surgeon such that the surgeon is able to manipulatethe images without touching a keyboard, monitor or touchscreen, therebykeeping hands free and sterile for the surgery. The 3 D holographicimage is viewable with the naked eye rather than through headgear,thereby reducing eyestrain and metal fatigue to the surgeon.

As a consequence, the surgery is performed with substantially reducedimpact on the patient at the intervention site and with a reduced riskof contamination at the intervention site.

Example 4. The surgeon performs a surgery on a patient as in Example 1,Example 2 or Example 3. The surgery is performed remotely, such that thespinal expert surgeon is not present during the procedure. Nonetheless,the surgeon is able to direct the surgery remotely with minimal relianceon any attending health personnel who are present.

Example 5. Medical visualization software. Xenco medical visualizationsoftware allows for the loading of DICOM medical image data such as CTand MRI scans from a drive or directly from PACS in a hospital's networkfor segmentation, registration, and visualization. The Xenco Medicalsoftware translates the 2D slices of the DICOM into a volumetric 3Drendering so that the user can see both the 2D view of the patient'sanatomy as well as a 3D view. The Xenco Medical software is alsopre-loaded with digital files of Xenco Medical's spinal implantsincluding Xenco Medical pedicle screws, cervical and lumbar interbodies,as well as cervical and lumber plates and corresponding screws. The usercan pick specific Xenco Medical implants and specific sizes from adropdown menu and load them into the viewer with the patient anatomy toevaluate various implants relative to each patient's anatomy. Inneighboring windows, the user sees both the 3D view of the implants andanatomy as well as a 2D view of the implants and anatomy simultaneously.The software is integrated with the Polaris Vega VT Position Sensor byNorthern Digital for surgical navigation, a Holographic Light FieldDisplay for holographic visualization, and a Leap Motion controller fortouch-free manipulation of the anatomy.

Example 6. Passive Position Sensors—Polaris Vega VT Position Sensor andpassive sensors by Northern Digital. The Polaris Vega VT Position Sensorby Northern Digital allows for the real-time, intraoperative tracking ofsurgical instruments by tracking passive sensors which are attached toeach instrument via a clip. There is a reference bracket with passivemarkers, which is placed on the patient. An attachment with passivesensors is placed on each instrument and they are calibrated andregistered before beginning. The Polaris sensor communicates with thesoftware by using the pre-operative CT or MRI of the patient as areference. As the tools are moved in the field of view of the sensor,the position of the instruments, relative to the anatomy of the patient,are displayed in the software. This is especially important in gaugingthe depth of the surgical instrument relative to the patient's anatomy.The user is able to see the instruments relative to both the patient's2D anatomical visualization and 3D anatomical visualization. This isalso viewable on the holographic light field display, which is aseparate monitor.

Example 7. Holographic Light Field display Sony Spatial Reality Display.The holographic light field display by Sony, known as their SpatialReality Display, tracks the movement of an individual user and adjuststhe 3D image in the light field based on the position of the eyes tocreate a hologram. The anatomical data from the Xenco Medical software,including instrument position during surgical navigation, can be viewedthrough the Sony Spatial Reality Display, giving the user a sense of thepatient's actual anatomy hovering in air. The Sony Spatial RealityDisplay currently has a diagonal length of 15.6 inches. The user canmanipulate the holographic anatomy through hand gestures in the airusing a Leap Motion controller, a motion sensing device capable oftracking precise hand and finger movements made in the air above thecontroller.

Example 8. Holographic Light Field display. Looking Glass Factory 8KDisplay. The holographic light field display by Looking Glass Factoryknown as its Looking Glass 8K Display also produces holographic imagesbut does not tracking eye movement. It also used light field displaytechnology and currently has a diagonal length of 32 inches. Theanatomical data from the Xenco Medical software, including the surgicalnavigation, can be viewed through the Sony Spatial Reality Display,giving the user a sense of the patient's actual anatomy hovering in air.The user can manipulate the holographic anatomy through hand gestures inthe air using a Leap Motion controller, a motion sensing device capableof tracking precise hand and finger movements made in the air above thecontroller.

Example 9. Leap Motion Controller. The leap motion controller is amotion sensing device able to track objects with sub-millimeteraccuracy. The controller allows the user to manipulate patient anatomywith hand gestures made in the air over the controller.

The following is a listing of claims related to the disclosure. It isunderstood that other claims are also supported by the disclosure, andthat claims may change, and different claims may be recited through thecourse of prosecution, consistent with rights granted through thisapplication

We claim:
 1. A method of surgical procedure instruction, comprising: i)presenting, to an audience, a) a patient region image, said patientregion image generated from a patient region dataset; and b) a surgicalprocedure instrument image, said surgical procedure instrument imagegenerated from a surgical instrument dataset; ii) adjusting the patientregion image, such that the patient region image reflects an impact ofan actual surgical procedure instrument on an actual patient imageregion; wherein the adjusting the patient region image is generatedwithout replacing the patient region dataset.
 2. The method of claim 1,wherein the audience comprises a medical student.
 3. The method of claim1, wherein the audience comprises a medical practitioner.
 4. The methodof claim 1, wherein the presenting comprises projecting a holographicthree dimensional depiction of the patient region image.
 5. The methodof claim 4, wherein the presenting comprises projecting a holographicthree dimensional depiction of the surgical procedure instrument.
 6. Themethod of claim 1, wherein the presenting comprises projecting a twodimensional depiction of the patient region image.
 7. The method ofclaim 4, wherein the presenting comprises projecting a two dimensionaldepiction of the surgical procedure instrument.
 8. The method of claim1, wherein the patient image region is malleable.
 9. The method of claim8, wherein the patient image region may be manipulated through ahands-free interaction.
 10. The method of claim 1, wherein thepresenting is concurrent with performing the surgical procedure.
 11. Themethod of claim 1, wherein the presenting is subsequent to performingthe surgical procedure.
 12. The method of claim 1, wherein thepresenting is prior to performing the surgical procedure.
 13. The methodof claim 10, wherein the actual patient region is labeled using at leastone patient region positional marker.
 14. The method of claim 13,wherein the at least one patient region positional marker signals atleast one change in the actual patient region posture.
 15. The method ofclaim 10, wherein the actual surgical instrument is labeled using atleast one surgical instrument positional marker.
 16. The method of claim15, wherein the at least one surgical instrument positional markersignals at least one change in the actual surgical instrument posture.17. The method of claim 1, wherein the presenting is remote to thesurgical procedure.
 18. The method of claim 1, wherein the presentingcomprises transmission of data.
 19. The method of claim 1, wherein thepresenting does not comprise use of headgear.
 20. The method of claim 1,wherein presenting comprises selectively changing a color or shade ofthe patient region image or surgical instrument image or both patientregion image and surgical instrument image.